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Executive Summary 


Introduction 


In 2015, the Research and Planning Group for California Community Colleges (RP Group) was 
contracted by the Joyce/Cabrillo Foundation to assess the long-term impact of the Academy for 
College Excellence (ACE) on students who had participated in the program between fall 2003 
and spring 2014 at three California Community Colleges: Berkeley City, Cabrillo, and Hartnell 
Colleges. To accomplish this goal, the RP Group designed and implemented a mixed-methods 
research study, gathering both qualitative and quantitative data to analyze the effect of 
participating in ACE in both students’ personal and professional lives. 


This report summarizes the key qualitative and quantitative research findings highlighting the 
ways in which the Academy for College Excellence (ACE) helped build the capacity of individuals 
facing numerous obstacles to success thrive as students, participants in the workforce, and 
members of the community.? 


Overview of the Academy of College Excellence 


The main goal of ACE is to prepare students to navigate and succeed in college, work, and life. 
In particular, ACE is designed to help individuals who have historically faced a wide range of 
challenges to academic and career success—such as underrepresented groups, foster youth, 
veterans—achieve their educational and employment goals. The program is designed to foster 
intrinsic engagement and motivation among community college students, providing support 
through curriculum and instruction that holistically addresses students’ needs (Navarro, 2017; 
Navarro & Hayward, 2014). 


ACE offers a flexible model that can be scaled, sustained, and replicated to serve different types 
of learners, such as basic skills students, career/technical education (CTE) students, as well as 
college-ready students. Many colleges have adapted the ACE model to meet the particular 
needs of their student populations, as well as align with the resources available at the 
institution. Figure 1 on the following page illustrates the components of the fully implemented 
ACE model. An individual college’s ACE program may contain some or all of these components, 
with the exception of the mandatory Foundations of Leadership Course. Prior to beginning their 
academic coursework, all ACE students complete this intensive two-week college-level course, 
which prepares students for academic engagement and professional careers by focusing on the 
development of professional skills targeting affective mindsets and behaviors associated with 
student success in both career and school (Asera & Navarro, 2013). 


1ACE has been the subject of several longitudinal research studies: 
http://academyforcollegeexcellence.org/reports-on-ace/ 
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Figure 1. The ACE Model 


Accelerated Pre-Stats 
English Math 
ACE Foundation Course ACE Team \ 
Two-Week Intensive —»| Self-Management Project Based 
Affective Orientation Course ae Social Justice 
i: R h 
3 semester credits ; esearc Wejcar 
2 semester credits Course . 
ra Planning 
Computer 
Applications 


Source: Karandjeff and Cooper (2013) 


Evaluation Methodology 


Qualitative data for this evaluation was collected via telephone surveys with 435 individuals 
who had completed ACE at some point between 2003 and 2014 at one of three colleges that 
host the ACE program: Cabrillo College, Berkeley City College, and Hartnell College. The 
telephone survey was a 30-question instrument designed to capture former students’ 
impressions of the ACE program, with an emphasis on how specific program components may 
have impacted students’ academic, career, and personal outcomes. In particular, the survey 
explored how the ACE personal success skills (see Table 1) have affected participants’ 
employability, competitiveness in the workplace, emotional well-being, community ties, and 
capacity to develop and realize academic and professional goals. 


Table 1. ACE Personal Success Skills 


Personal Success Skill Examples 


Realizing one’s own influence over life and personal success 

Considering the consequences of one’s actions 

2. Self-awareness and self- e Understanding bioreactions, such as fight, flight, freeze, or 
discipline appease 

e Identifying, grasping, and aligning needs/concerns of individuals 
with the goals of a project team or organization practiced by 


1. Purpose and direction 


3. Communicating with 


others 
leaders 
4. Working styles e Understanding one’s own and others’ working styles 
5. Social justice research e Cultivating critical thinking through the use of research methods 


Quantitative research complemented the telephone survey with an in-depth analysis of ACE 
participants’ educational and employment outcomes and comparison of those outcomes to a 
demographically similar group of students who did not enroll in ACE. Looking at just under 
3,000 former ACE students from Cabrillo, Berkeley City, and Hartnell Colleges, the researchers 
compared key academic outcomes to a similar group of non-ACE participants as well as 
whether median annual wage differences existed over time between ACE and non-ACE 
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participants. Student-level information from each college, as well as data from the California 
Community Colleges Chancellor's Office Management Information System (MIS) and the 
California Employment Development Department 


(EDD) was gathered to answer these research ACE Student Characteristics* 


questions. 
GENDER 


Key Findings Y 56% male 


¥ 44% f | 
This section highlights some of the key findings ae 
from both the qualitative and quantitative ETHNICITY 
research conducted. 63% Hispanic 
A Population under Significant Strain te Ne 


7% African-American 
As alluded to earlier, ACE students typically face a 


wide range of barriers to success in educational 


11% Other / Unknown 


settings. An examination of a subset of the RISK FACTORS 
students tracked for this evaluation revealed Nearly one-fourth (26%) faced multiple risk 
almost 20 risk factors—such as child abuse, factors, most commonly: 
mental health condition, medical condition, Y Government assistance (40%) 
unstable housing, history of being in foster care, English as a second language (39%) 
currently or previously on probation, homeless, Unstable home (38%) 
and gang association—in these students’ lives, Previous probation (28%) 
and the majority of students were juggling 
multiple risk factors (Farr, Rotermund, Radwin, sae) 
Robles, and Choy, 2014). ¥ Gang association (25%) 

*Demographic data reflect community college 
These substantial obstacles to success must be records and self-reporting from over 2,500 non- 
taken into consideration when assessing ACE CTE ACE students examined during the program 
participants’ academic and earnings outcomes in evaluation. 


comparison to their demographically similar peers. In many cases, simply keeping pace with 
other community college students who are not facing such an onslaught of barriers is a 
powerful achievement in and of itself. 


ACE Participants’ Academic Outcomes 


To assess the academic progress of former ACE participants, the RP Group looked at certain key 
academic milestones, including successful completion of transfer-level math and English 
courses, nursing and pre-nursing coursework, achievement of 30 transfer-level units, 
completion of a certificate or degree, and transfer to a four-year institution. Analysis of ACE 
participants’ achievement of these milestones in comparison to a matched group of similar 
students revealed the following: 


Vv The greatest positive academic outcomes were found among students who completed 


all the courses in the ACE program that include accelerated English, especially in those 
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programs that were most compliant with the full ACE model. Most times, these 
students were at least twice as likely as students in the control group to earn an award, 
complete a transfer-level English or math course, and earn 30 units within three years 
of enrolling in ACE. 


A greater percentage of ACE nursing students graduated within three and six years 
than their pre-ACE nursing peers. 


Looking at all academic outcomes over a six-year period, the researchers found no 
statistically significant differences between ACE students and their peers in the control 
group (with the exception of completion of a transfer-level English and transfer to a 
four-year institution). Please note: This result does not include students who 
participated in ACE as part of a CTE nursing program, which incorporated ACE into its 
existing cohort-based model. 


Underrepresented minorities who participated in ACE students completed 
degrees/certificates and transferred to four-year institutions at similar rates to their 
matched peers. 


While these findings may initially seem modest, given the substantial array of risk factors faced 


by ACE students, it is remarkable that over the long-term, their academic achievements 


equaled, or in some cases even exceeded those of other students who began the academic 


“race” much closer to the finish line. 


ACE Participants’ Earnings Outcomes 


Data from EDD was used to track the wages of ACE participants and the control group of 
students over time. Key findings from this analysis include the following: 


v Even though the annual mean wages for the control group were higher overall and 


over time, ACE participants had a much higher wage gain (159% to 664%) over time 
than students in the control group (107%). 


Both males of color and underrepresented minority students in general who 
participated in ACE started to catch up to the control group’s earnings levels by the 
sixth year. In fact, underrepresented minority ACE students eventually exceeded the 
annual median wages of the control group by almost $1,000. 


ACE nursing students earned significantly higher wages over time compared to their 
matched control group. The wage differences over time between ACE nursing students 
and their matched peers were greater than it was for non-ACE nursing students and 


their peers. 


As with ACE students’ academic outcomes, these earnings outcomes may at first seem 


unremarkable. However, when examined in the context of all of the challenges faced by ACE 
participants, simply being able to match their more-advantaged peers in terms of earnings is a 
significant accomplishment. Moreover, the fact that some ACE participants achieved wages that 


exceeded those of their peers is a powerful endorsement of the ACE program. 
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Impact of Developing Personal Success Skills 


As described earlier, the telephone survey asked over 400 former ACE participants how the 
personal success skills they developed during the ACE program affected their lives going 
forward. An analysis of survey results revealed the following: 


v¥ The development of purpose and direction had the most positive influence on students’ 
lives, particularly in the area of work. 


vY Learning self-awareness and self-discipline enabled ACE students to pause and regain 
perspective in both job-related and personal interactions. 


Y The cultivation of collaborative leadership and communication skills empowered ACE 
participants to break down communication barriers in personal and professional 
situations. 


¥ Understanding their own and others’ working styles proved critical to successful job 
performance and personal relationships. 


Y Most respondents “agreed” or “strongly agreed” that personal success skills had a 
positive influence on their work and personal lives (75% and 92%, respectively). 


Conclusion 


The findings drawn from this comprehensive quantitative and qualitative evaluation of the 
impact of the ACE program suggest that ACE has a positive effect on students’ academic, 
career, and personal success. The extensive telephone interviews conducted indicate that ACE 
supports students’ academic and social integration in higher education and delivers psycho- 
social supports for underserved and underrepresented college students. 


Furthermore, for historically underrepresented, first-generation, and “at-risk” students, ACE 
promotes postsecondary education as a tool for personal and economic success and facilitates 
participants’ effective navigation of and success in the college/professional environment. 
Finally, findings indicate that participation in ACE builds students’ ability to complete transfer- 
level coursework, achieve educational awards, and increase earnings over time. 
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Introduction 


For More Information... 


In 2015, the Research and Planning Group for California ; 

; i h For more information on the RP Group’s 
Community Colleges (RP Group) was contracted by the foneiicicimn ede node uray onihe 
Joyce/Cabrillo Foundation to assess the long-term Academy for College Excellence, visit 
impact of the Academy for College Excellence (ACE) on http://rpgroup.org/All- 
students who had participated in the program between Projects/ArticleType/ArticleView/ArticlelD/152 


fall 2003 and spring 2014 at three California Community  (Upgiseiilsle AUT Cs a ets co ae 
Coll - Berkeley Ci brill dH II Coll Mack (rpurnell@rpgroup.org) and Alyssa Nguyen 
olleges: Berkeley City, Cabrillo, and Hartnell Colleges. Ano Henn UNO 


To accomplish this goal, the RP Group designed and 
implemented a mixed-methods research study, OV aileh fe nycelrafenavel Gla tt esCecTere3 Fels 
gathering both qualitative and quantitative data to ROICBC EAC UEMICE) Visit 

xe ee : http://academyforcollegeexcellence.org/. 
analyze the effect of participating in ACE in both 


students’ personal and professional lives. 


This report summarizes the key qualitative and quantitative research findings highlighting the 
ways in which the Academy for College Excellence (ACE) helped build the capacity of individuals 
facing numerous obstacles to success to thrive as students, participants in the workforce, and 
members of the community. 


In This Report 


This report begins with an overview of the ACE program, including its purpose, its core 
components, and the preparation faculty receive to deliver its unique pedagogy. Next, the 
research methodology employed in this mixed-methods evaluation is summarized, and a 
description of ACE participants’ demographics and primary challenges is provided. The 
following section describes key findings from the evaluation, examining ACE participants’ 
academic and earnings outcomes in comparison to a matched group of similar students. 
Following, the report delves into the impact of the personal success skills taught in the ACE 
program on participants’ personal and professional lives. Finally, a conclusion offers some final 
thoughts on the overall long-term impact of ACE on participating students, as well as the 
limitations of this study and potential avenues for future research. 


Overview of the Academy for College 
Excellence 


Purpose of the ACE Program 

Launched in 2002!, the main goal of ACE is to prepare students to navigate and succeed in 
college, work, and life. In particular, ACE is designed to help individuals who have historically 
1 The research in the ACE design and the initial pilots was conducted in 2002. The first cohort was in fall of 2003. 
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faced a wide range of challenges to academic and career success—such as underrepresented 
groups, foster youth, veterans, and more—achieve their educational and employment goals. 
The program is designed to foster intrinsic engagement and motivation among community 
college students, providing support through curriculum and instruction that holistically 
addresses students’ needs (Navarro, 2017; Navarro & Hayward, 2014). 


ACE offers a flexible model that can be scaled, sustained, and replicated to serve different types 
of learners, such as basic skills students, career/technical education (CTE) students, as well as 
college-ready students. Many colleges have adapted the ACE model to meet the particular 
needs of their student populations, as well as align with the resources available at the 
institution. An individual college’s ACE program may contain some or all of these components, 
with the exception of the mandatory Foundations of Leadership Course (Foundation Course). 


The ACE Model 


Figure 1 below page illustrates the components of the fully implemented ACE model. 


Figure 1. The ACE Model 


Accelerated Pre-Stats 
English Math 
ACE Foundation Course ACE Team \ 
Two-Week Intensive => Self-Management Project Based 
Affective Orientation Course z5 Social Justice 


Research 
Course 


3 semester credits 


*\ career 


Planning 


2 semester credits A 


Computer 
Applications 


Source: Karandjeff and Cooper (2013) 


Unique ACE Courses 


The ACE model includes several unique courses that have been specifically designed to cultivate 
students’ engagement and motivation as well as equip them with the skills and knowledge 
necessary to succeed in future academic and employment pursuits. Key aspects of the 
Foundations of Leadership, Social Justice Research, and Team Self-Management courses are 
described below. 


FOUNDATIONS OF LEADERSHIP COURSE 


Prior to beginning their academic coursework, all ACE students complete the intensive, two- 
week, college-level Foundation Course, which focuses on the development of professional skills 
targeting affective mindsets and behaviors associated with student success in both career and 
school (Asera & Navarro, 2013). Specifically, the Foundation Course is built around the 
following three learning objectives: 


Academy for College Excellence: Mixed-Methods Analysis of Long-Term Outcomes 
The RP Group | March 2018 | Page 11 


1. Analyze, synthesize, and apply models and stages of dynamic leadership and principles 
of team self-management 

2. Analyze and critique the industrial model of education and its effects on students’ 
performance in US public schools 

3. Analyze, develop, and practice communication skills for leadership in academic, 
professional, and grassroots contexts 


SOCIAL JUSTICE RESEARCH COURSE 


ACE’s Social Justice Research Course (SJRC) is designed to engage students in primary research 
to identify ways to address a social or community issue that is of relevance to them. The course 
allows students to see themselves as change agents who can help unpack and solve issues that 
may affect them and their communities. Students work in cooperative teams to investigate a 
compelling social issue, develop research questions, design a survey, collect 150 surveys, 
analyze collected survey data, and highlight descriptive statistics. Students’ findings are then 
used to develop an action plan to inform and address community needs. ACE students create a 
PowerPoint slide deck that summarizes their research and action plan and deliver a 
presentation on their work to an audience of ACE stakeholders, such as community college 
administrators, students and faculty. 


TEAM SELF-MANAGEMENT COURSE 


The related Team-Self Management course builds on a learning action plan that students 
complete in the Foundation Course by having students reflect on their experiences and goals. 
This class provides opportunities for students to strengthen peer-to-peer networks, increase 
their understanding of the behaviors necessary for college success through reflection, and 
improve their ability to plan for and manage challenges that may hinder their educational 
progress. Course activities inspire students to be more confident as students and professionals, 
learn how to self-regulate their emotions, focus their attention, and positively and effectively 
communicate with others (Karandjeff & Cooper, 2013). 


Preparing Faculty to Teach in the ACE Program 


To effectively deliver the unique and innovative coursework of the ACE program, community 
college faculty need to employ pedagogical techniques that differ from the traditional methods 
with which most educators are familiar. To prepare community college faculty for this 
endeavor, ACE requires that faculty participate in a Five-Day Experiential Learning Institute 
(FELI) and complete a practicum, teaching students with the guidance and support of a master 
teacher. The FELI, a practicum, and additional trainings along with curriculum kits prepare 
newly-minted ACE faculty to offer students an educational experience in which “curriculum and 
pedagogy are intertwined” and both faculty members’ and students’ personal lives and 
perspectives help shape and inform the learning environment (Asera & Navarro, 2013, p. 2). 
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Evaluation of the ACE Program 


Purpose of the Evaluation 


This research study builds on and adds to the numerous research and evaluation studies that 
have found positive academic outcomes for ACE participants, such as completion of a transfer- 
level English and math (Farr, Rostermund, Radwin, and Robles, 2012; Jenkins, Zeidenberg, 
Wachen, and Hayward, 2009; RTI, 2014). This investigation, however, seeks to understand the 
long-term impacts of the ACE model on participating students’ academic, personal, and 
professional outcomes, such as degree/certificate completion, the impact of the personal 
success skills developed in the ACE program, and wage attainment. 


For the purpose of this mixed-methods investigation, ACE participants were operationally 
defined as students who were enrolled in an ACE Foundation course between fall 2003 through 
spring 2014 at one of three California Community Colleges: Berkeley City College, Cabrillo 
College, and Hartnell College. 


Research Methodology 


In consultation with the Academy for College Excellence, the Joyce Foundation, and workforce 
development experts, the RP Group led the design and execution of a mixed-methods research 
study to understand and advance the long-term impacts of the ACE program model. This 
research included two parts: (1) a telephone survey with former ACE students (qualitative), and 
(2) a statistical analysis of ACE participation related to educational trajectory and employment 
outcomes (quantitative). For the purpose of this report, telephone surveys may also be 
referenced as “interviews.” 


Qualitative Research Methods 


Qualitative data for this evaluation was collected via telephone surveys with 435 individuals 
who had completed ACE at some point between 2003 and 2014 at one of three colleges that 
host the ACE program: Cabrillo College, Berkeley City College, and Hartnell College. The 
telephone survey was a 30-question instrument designed to capture former students’ 
impressions of the ACE program, with an emphasis on how specific program components may 
have impacted students’ academic, career, and personal outcomes. In particular, the survey 
explored how the personal success skills taught in the ACE program have affected participants 
employability, competitiveness in the workplace, emotional well-being, community ties, and 
capacity to develop and realize academic and professional goals. 


, 


? Excludes students who were enrolled in new allied health programs that were too small to include in the CTE 
cohort (Medical Assisting at Cabrillo and Respiratory Health at Hartnell). 
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Quantitative Research Methods 


Looking at just under 3,000 former ACE students from Cabrillo, Berkeley City, and Hartnell 
Colleges, the researchers compared key academic outcomes to a statistically-equivalent group 
of non-ACE participants (using a 1:1 propensity score matching analysis),? as well as whether 
median annual wage differences existed over time between ACE and non-ACE participants. 
Student-level information from each college, as well as data from the California Community 
Colleges Chancellor's Office Management Information System (MIS) and the California 
Employment Development Department (EDD) was gathered to answer these research 
questions. 


PROPENSITY SCORE ANALYSES 


Students from Cabrillo College, Berkeley City College, and Hartnell College who had participated 
in ACE were compared to a matched non-participant peer group using a one-to-one match 
propensity score matching analysis. As students’ participation in the ACE program is voluntary 
and not at random, any results from an analysis of participant outcomes may be influenced by 
selection bias. That is, ACE participants may not be representative of all college students, and 
so a comparison of ACE participants with nonparticipants may be influenced, or biased, by 
some unobservable trait(s) of students in the ACE participation groups. One way of addressing 
the problem of selection bias is through a quasi-experimental approach that uses propensity 
scores to form a comparison, or control, group that resembles the ACE model participant, or 
treatment group (Rosenbaum & Rubin, 1983; Wooldridge, 2012). 


For this study, we used a comprehensive list of 22 background variables that could vary 
between ACE and non-ACE participants. These 22 variables cover demographic characteristics, 
socioeconomic status, and academic history and goals (see Appendix B, Table B1). However, it is 
important to note that we could not match ACE peers on risk factors such as prior 
incarceration, drug and alcohol use, and mental health issues because similar risk factor data 
are not available for the general student population, even though it is available for the ACE 
students. Although, the comparison group created through propensity score matching will be 
identical in many ways (e.g., race, ethnicity, GPA, English/math classes taken), the presence of 
risk behaviors could still be a differentiating factor between the control group and the ACE 
students. 


The logistic regressions resulted in a propensity score for each student that represents an 
individual's likelihood, or propensity, of participating in the ACE model based on the identified 
22 background variables. After matching, we evaluated how well the models performed by 
examining the differences in predictor variables after matching and the percent reduction in 
bias from before matching. Essentially all differences were non-significant after matching (See 
Appendix B, Figures B1-B5). However, please note that this quasi-experimental approach can 
only account for the observed differences and does not account for other confounding factors. 
That is, this adjustment does not entirely eliminate the problem of selection bias, but does 


3 See Appendix B: Analyses, Table B1 for list of variables used in Propensity Score Analyses. 
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increase the confidence that the results are not entirely due to preexisting differences between 
the treatment and control groups in regards to the selected variables. 


Who Are ACE Students? 


Participant Demographics 


Table 1 below provides descriptive statistics of basic demographic information for students 
enrolled in ACE. Based on the data, a slight majority of ACE students were male (545%), while a 
strong majority were and of Hispanic backgrounds (60%). 


Table 1. Descriptive Statistics for ACE Participants in the Quantitative Study 


Number of Percentage of 


Characteristic “ - 
Participants Participants 


Gender 
Male 1,595 53.5% 
Female 1,387 46.5% 
Ethnicity 
Hispanic 1,790 60.0% 
White 605 20.3% 
African-American/Black 186 6.2% 
Unknown 183 6.1% 
Other 121 4.1% 
Asian 97 3.3% 
Total 2,982 100.0% 


A Population Under Significant Strain 


Since the ACE program seeks to engage and support students who may face unique challenges 
and hardships that could hinder their academic journeys, the program’s intake form asks 
students to report on risk factors that inform program design and implementation. As alluded 
to earlier, ACE students typically face a wide range of barriers to success in educational settings. 
An examination of a subset of the students tracked for this evaluation revealed almost 20 risk 
factors—such as child abuse, mental health condition, medical condition, unstable housing, 
history of being in foster care, currently or previously on probation, homeless, and gang 
association—in these students’ lives, and the majority of students were juggling multiple risk 
factors (Farr, Rotermund, Radwin, Robles, and Choy, 2014). 


In seeking to transform the college experience for underprepared and vulnerable college 
students, ACE programs intentionally recruit opportunity youth — young adults aged 16-24 
years who are neither in school nor working. It is not uncommon for students recruited into an 
ACE program to be at-risk or exposed to high-risk factors, such as being homeless or facing 
domestic violence. Using self-reported information collected from the ACE Bridge Survey, Table 
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2 on the next page provides information about how many of the ACE participants had risk 
factors that are known to be barriers to success (e.g., history of being arrested, child abuse). 
The four factors most commonly reported, from highest to lowest, were: receives government 
assistance, from an unstable home, has been on probation, and were/are associated with 
gangs. On average, ACE students self-reported four to five risk factors. 


Table 2. Percentage of ACE Participants with Various Risk Factors (N = 936) 


Risk Factor ACE Participant Counts Percent of All Participants 
Child abuse (H) 119 12.7% 
Parent with dependent children (A) 212 22.7% 
Working while in school (A) 205 21.9% 
Mental condition (H) 42 4.5% 
Medical condition (H) 60 6.4% 
Foster care history (H) 76 8.1% 
Receives government assistance (A) 379 40.5%: 
Unstable home (H) 353 37.7% 
Has been on probation (H) 258 27.6% 
Currently on probation (H) 150 16.0% 
Gang association (H) 231 24.7% 
Homeless (H) 217 23.2% 


Note: A = At-risk factor and H = High risk factor. Numbers and percentages are for non-nursing students only. 
Student risk characteristics were collected from the ACE Bridge Survey where participant intake forms were merged 
with the student records obtained for this study. A total of 936 of the 2,982 students (31.4% match rate) were 
successfully matched with the risk statistics file. 


Key Evaluation Findings 


This section highlights key quantitative and qualitative findings that emerged from the ACE 
evaluation. To fully understand the implications of these data, it is essential to keep in mind the 
substantial obstacles to success faced by ACE participants. From issues such as severe financial 
difficulties to language barriers, housing instability, legal issues, and gang association, the 
majority of ACE participants were forced to start the “race” to success already at a significant 
disadvantage when compared to their peers. 


As such, these profound challenges must be taken into consideration when assessing ACE 
participants’ academic and earnings outcomes in comparison to their demographically similar 
peers. In many cases, simply keeping pace with other community college students who are not 
facing such an onslaught of barriers is a powerful achievement in and of itself. 
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ACE Participants’ Academic Outcomes 


To assess the academic progress of former ACE participants, the RP Group looked at the 
following six academic milestones: 


Successful completion of transfer-level math 

Successful completion of transfer-level English 

Completion of 30 transfer-level units 

Completion of a degree or certificate 

Transfer to a four-year institution 

Transfer preparedness (earned 60+ transferable units with a 2.0+ GPA) 


Oo Bow Ne 


Overall Academic Achievement of ACE Participants 


A logistic regression was run comparing the six academic outcomes for all ACE participants (any 
student who enrolled in the Foundation Course) and the matched control group (see Appendix 
C, Table C1 for detailed results). Results from these analyses revealed that a larger percentage 
of the matched control group achieved the outcomes than the ACE participants. However, the 
achievement percentages across all outcomes for ACE participants were very close to their 
matched peers. More importantly, academic outcome comparisons within six years actually 
found no statistically significant differences between ACE participants and the matched control 
groups for three out of the six targeted academic outcomes: completion of award, successful 
completion of transfer-level math, and completion of 30 transferable units. 


While these findings may initially seem modest, given the substantial array of risk factors faced 
by ACE students, it is remarkable that over the long-term, their academic achievements 
equaled or came close to matching those of other students whose educational pathway was 
fraught with many fewer obstacles and challenges. 


The RP Group also conducted a statistical comparison of the six academic outcomes for under- 
represented minorities (URM)—specifically, African American/Black, Hispanic, or Asian students 
who participated in ACE and students with the same ethnic background from the control group. 
Results from these analyses found trends that were similar to the overall ACE students and 
their matched control group, where the control group completed outcomes at higher rates 
within three years, but after six years, ACE students started catching up with their peers. Most 
notable however for URM ACE students is that after six years, URM ACE students completed 
degrees/certificates and transferred to four-year institutions at comparable rates to their 
matched control group (see Appendix C, Table C2). 


Academic Outcomes by ACE Program Variation‘ 

As described earlier, ACE offers a flexible model that colleges can implement based on 
availability of resources, student and faculty interest, and numerous other factors. While all 
4 For a complete list of the courses that comprise each of ACE’s cohort variations, see the Appendix A. 
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ACE programs must include the Foundation Course, colleges can choose to offer the full ACE 
program—including social justice research, accelerated English, team self-management, 
movement (physical education), and career planning—or only some of those components. 


ACADEMIC ACHIEVEMENTS OF NON-CTE ACE STUDENTS 


An examination of the academic outcomes of students who participated in different variations 
of the ACE model reveals the powerful impact of each component of the ACE program. Based 
on the analyses, it does appear there are differential impacts on students’ academic outcomes 
depending on the ACE program design (see Appendix C, Tables C3-6). In fact, participants in ACE 
programs that were most compliant with the complete ACE program model? had the best 
outcomes compared to their peers in less comprehensive versions of ACE. Moreover, when 
these students’ academic outcomes were compared to the matched control group, the 
evaluation showed that they were: 


¥ 2.1 times more likely than control group students to earn an award within six years; 
¥ 2.4times more likely to complete a transfer-level English course within three years; 

¥ 2.0 times more likely to complete a transfer-level math course within three years; and 
¥ 2.7 times more likely to earn 30 units within three years. 


ACADEMIC OUTCOMES FOR ACE STUDENTS IN CTE PROGRAMS 


Furthermore, the ACE program had a particularly strong impact on the academic outcomes of 
students in one career/technical education (CTE) program (in this study, nursing students).° As 
illustrated in Table 4, a statistically significant larger percentage of nursing students who had 
begun participation in an ACE model between 2009 and 2014 (“ACE nursing students”) 
completed an award within three- and six-years than those who had not participated in ACE 
and began the nursing program between 2003 and 2009 (“Pre-ACE nursing students”).’ 
Proportionally, twice as many ACE nursing students earned a degree/certificate within three 
years than pre-ACE nursing students. 


5 Cohort variation 7 is most compliant with the ACE model, including the Foundation Course, Team Self- 
Management, Social Justice Research, Accelerated English, Movement, and Career Development. See the Appendix 
for a list of all ACE cohort variations. 

5 Only nursing was included because the other two comparable CTE programs that embedded a Foundation course 
had sample sizes that were too small to include in the CTE cohort (Medical Assisting at Cabrillo and Respiratory 
Health at Hartnell). 

7 Since fall 2009, Hartnell embedded the ACE Foundation course in the first year for all cohorts in the program. 
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Table 4. Comparison of Award Attainment for Pre-ACE and ACE Nursing Students 


Gompletienole Pre-ACE Nursing ACE Nursing Students 


Degree or Difference 
a Students (N = 236) (N = 499) 
Certificate 
P Poi 
Timeframe Percentage of Students Percentage of Students eieetees aut 
Difference 

Within 3 years* 27.7% 64.3% 36.6% 
Within 6 years* 43.4% 67.6% 24.2% 


Notes. p < 0.05. Only students enrolled in academic years where at least three- and six-year rates could be 
calculated were included in the analysis. The cohort for three-year rates were Pre-ACE nursing students enrolled 
between fall 2003 and spring 2009 (n = 459) and ACE nursing students enrolled between fall 2009 and spring 2013 
(n = 157). The cohort for six-year rates were Pre-ACE nursing students enrolled between fall 2003 and spring 2009 
(n = 459) and ACE nursing students enrolled between fall 2009 and spring 2010 (n = 37). 


Earnings Outcomes 


In order to examine the impact that student participation in ACE may have on future earnings, 
the annual mean wages of ACE students were compared to the annual mean wages of their 
matched peers one year before and up to six years after the ACE term of enrollment. Annual 
mean wage comparisons between groups of individuals who did or did not participate in ACE 
were made with several t-tests (assuming unequal variances) that compared the annual mean 
wages of ACE participants to the overall control group. EDD match rates for each of the wage 
metric produced match rates ranging from 45% to 100% (see Appendix D). In general, match 
rates appear to increase over time, which is not surprising, assuming that students are 
becoming gainfully employed after they complete their academic studies, but results should be 
interpreted with caution as the match rates only suggest that no wages were found, but not 
why. The inability to match wages may be due to unmatched/invalid social security numbers or 
individuals being employed by employers who do not report wages into the state system, or 
unemployment. 


Earnings Change by ACE Model Variation 


NON-CAREER/TECHNICAL EDUCATION ACE VARIATIONS 


Overall, the annual mean wages of ACE students were lower than the annual mean wages of 
the matched control group, one year before and every year thereafter (see Figure 2). However, 
a closer examination of the data reveals that the wage gains for ACE students (change in wages 
from before ACE term vs. six years after) were double the wage gains experienced by the 
matched control group. ACE students had a 200% wage gain from one year before their ACE 
enrollment to six years after, while the matched control group only had a 107% wage gain in 
the same time points. 
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Figure 2. Comparison of Annual Mean Wages Over Time: ACE vs. Control Group 
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Note. See Appendix A for size of each group and Appendix D for match rates by year. 


To further explore the potential impact of the ACE program variation, wages were pooled for 
students who completed the various configurations of the ACE model. In order to explore any 
potential wage differences over time, the major ACE program variations (i.e., meta-levels) were 
pooled into the following model variations: (1) Accelerated English, (2) Un-Accelerated English, 
(3) Limited Resources, and (4) Foundation Only, Non-CTE (see Appendix A for model variation 
descriptions). In all ACE meta-level groups (except Foundation Only Non-CTE), students’ wages 
before ACE were lower than those of the control group and continued to be lower over time 
(see Figure 3 below). For students who completed a non-CTE Foundation course only, their 
wages starting three years post-ACE start to exceed the control group. However, these results 
should be interpreted with caution, as the higher annual mean salaries appear to be driven by 
outliers pulling the group average upwards (SD = 65,147). 
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Figure 3. Comparison of Pooled Annual Mean Wages Over Time by ACE Model 
Variation (Completers) 
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Note: See Appendix A for size of each group and Appendix D for match rates by year. 


However, when the evaluation team explored how ACE participants’ earnings changed over 
time for each ACE meta-level, the data revealed that ACE participants saw significantly more 
growth in their earnings over a six-year period than did their peers from the control group 
regardless of ACE program variation (see Table 6). Students in the control group had a wage 
gain of 107% from pre-ACE comparison term to six years post-ACE comparison term; while ACE 
students, regardless of program variation, had wage gains that ranged from 163% (Un- 
accelerated English) to 451% (non-CTE Foundation only). Information from the telephone 
interviews qualify these findings where it was found that a lower percentage of ACE 
participants worked in minimum wage occupations after their participation in ACE (22% post- 
ACE vs. 62% pre-ACE) and a higher percentage were employed after ACE (79% post-ACE vs. 54% 
pre-ACE). 
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Table 6. Average Mean Salary Change from One Year Before ACE to Six Years After 
by ACE Model Variation 


ACE Model Variation Average Salary Change 
Accelerated English 202% 
Un-Accelerated English 163% 
Limited Resources ACE 249% 
Foundation Course Only — Non-CTE ACE 451% 
Control Group 107% 


EARNINGS CHANGE AMONG UNDERREPRESENTED MINORITY GROUPS (NON-CTE) 


This growth over time was particularly evident for two historically-disadvantaged student 
populations: males of color (African American, American Indian, Hispanic, and Pacific Islander) 
and underrepresented minorities (URM) (African American, American Indian, Asian, Hispanic, 
and Pacific Islander). The change in these students’ annual mean wages before participation in 
ACE and six years after ACE was much greater than the wage growth experienced by their 
matched peers. By the sixth year following participation in ACE, the annual median wage 
change was 305% for URM students as a whole and 435% for males of color (see Figure 4 for 
Males of Color Comparisons). In comparison, the wage growth for both males of color and URM 
students in the control group was approximately 200%. 


Figure 4. Comparison of Annual Median Wages Over Time for Males of Color: 
Non-CTE ACE vs. Matched Peers 
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WAGE OUTCOMES FOR CTE STUDENTS 


As with academic outcomes, wage outcomes for CTE (nursing) students were examined 
separately and revealed a strong positive association with the inclusion of the ACE Foundation 
Course in the nursing program. Nursing students are unique compared to other students both 
in terms of motivation and potential labor market outcomes, which can vary considerably 
during different time periods. Therefore, to conduct the most accurate comparison possible, 
both Pre-ACE and ACE nursing students were matched with non-nursing students enrolled 
during the same time period. 


Figure 5 on the next page compares annual mean wages differences over time for the Pre-ACE 
nursing students and their matched peers with the annual mean wage differences for the ACE- 
nursing students and their matched peers. In general, both Pre-ACE and ACE nursing students 
started with lower annual mean wages than their matched peers, but over time, their annual 
mean wages started to exceed those of their peers. The difference in annual mean wages for 
the nursing students compared to their matched control group over time was significantly 
greater for ACE nursing students than it was for Pre-ACE nursing students. By the sixth year, 
ACE nursing students wages were $45,541 higher than their matched peers, while the wage 
differences for the Pre-ACE nursing students and their matched peers was only $8,875. These 
results suggest that the addition of the ACE Foundation course, may have had a positive impact 
on nursing students’ employability and wage earnings.? However, the increased wages could 
also be related to the higher percentage of ACE nursing students graduating within three years 
than the Pre-ACE nursing students. 


8 Although the wage comparisons revealed a statistically-significant difference across all seven years between Pre- 
ACE and ACE nursing students, it should be noted that increases in annual median salaries and unemployment 
rates in California over time may be affecting the labor market potential for these students. 
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Figure 5. Comparison of Difference in Annual Mean Wages for Pre-ACE and ACE 
Nursing Students versus Matched Peers Over Six Years 
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Impact of Developing Personal Success Skills 


ACE is comprehensively designed—including curricular content, pedagogy, and faculty 
training—to educate students who face multiple life setbacks that are likely to impede their 
academic achievement. In particular, the program focuses on imparting five personal success 
skills, described in Table 7 on the following page. 
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Table 7. ACE Personal Success Skills 


Personal Success Skill Examples 


Realizing one’s own influence over life and personal success 

Considering the consequences of one’s actions 

2. Self-awareness and self- Understanding how to work with their bioreactions, such as 
discipline fight, flight, freeze, or appease 

e Identifying, grasping, and aligning needs/concerns of individuals 
with the goals of a project team or organization practiced by 


1. Purpose and direction 


3. Communicating with 
others 


leaders 
4. Working styles e Understanding one’s own and others’ working styles 
5. Social justice research e =Cultivating critical thinking through the use of research methods 


These five sets of skills stem from ACE Founder Diego Navarro’s extensive academic and 
professional training stressing that affective or noncognitive skills are critical for today’s 21st 
century professionals. Affective factors are those that engage the emotions and feelings 
necessary for student success (Navarro, 2012). Decades of research on the affective domain in 
teaching and learning support the notion that changing mindsets and habits facilitates student 
achievement. While the first four personal success skills relate to feelings and behaviors, the 
fifth skill, Social Justice Research, enables students to both pinpoint a societal problem that is 
relevant and meaningful to their lives and apply the affective skills to address it with a team of 
students, and to accelerate the acquisition of English, math and critical thinking skills through 
project-based learning. 


As described earlier, the telephone survey asked over 400 former ACE participants how the 
personal success skills they developed during the ACE program affected their lives going 
forward. Ninety percent of students interviewed strongly agreed/agreed that the personal 
success skills influenced their professional lives and 86% strongly agreed/agreed that those 
skills influenced their personal lives (see Figure 6). 
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Figure 6. Summary of Students’ Assessment of Personal Success Skills Influence on 
Personal and Professional Lives 


Summary of Impact of Personal Success Skills - Overall 
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Further discussion of the impact of each of the five personal success skills is provided in the 
sections below. 


Skill 1: Purpose and Direction 


Following ACE, the skill set of purpose and direction reportedly had the most positive influence 
on students’ lives, particularly in the area of work. This personal success skill focuses on taking 
responsibility, developing agency, setting priorities, making decisions, and taking action to 
ensure personal success. One survey respondent, for example, asserted that purpose and 
direction meant, “| know what | have to do to develop the qualities I’m lacking.” 


During the interview, respondents were asked to cite their level of agreement with six basic 
principles of purpose and direction: 


Realizing my own influence over my life/the things that happen to me 
Thinking about the consequences of my actions before doing something 
Realizing that it is my choice whether | do well 

Being ready to learn 

Setting priorities 

Being prepared, organized, and knowing what | have to do 


am PwWNrP 


The majority of respondents agreed or strongly agreed that each of these six principles helped 
them find purpose and direction—in both their professional and personal lives (see Figure 7). 
However, respondents cited a stronger level of influence on their work life (as opposed to their 
personal life) with respect to all six of the statements. Specifically, respondents stated that 
“realizing that it is my choice whether | do well” had the strongest influence on their personal 
life, with 32% of respondents agreeing strongly with that statement. Almost 40% of 
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respondents also stated that ACE’s purpose and direction curricula helped them “think about 
the consequences of their actions before acting” on the job. 


Figure 7. Purpose and Direction Lessons Influencing Work and Personal Life 
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Skill 2: Self-Awareness and Self-Discipline 


Self-awareness and self-discipline, which foster emotional self-regulation and encourage 
students to trust others, were recognized as an important force in helping participants realize 
when they needed to pause and regain perspective in their work and personal interactions. In 
this domain of the curriculum, students learned the essential skill of understanding their 
responses to stress. Typically, ACE students face years of emotional strain and pressure prior to 
entering the ACE program. The personal success skills of self-awareness and self-discipline help 
participants cope with mental and physical stressors more effectively. 
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During the interview, respondents were asked to cite their level of agreement with five basic 
principles of self-awareness and self-discipline: 


Understanding when | am in bioreaction (fight, flight, freeze, or appease) 
Pausing in difficult situations without immediately reacting 

Realizing my personal capacity to deal with stress 

Judging people less 

Focusing and concentrating 


GB Gi 


The majority of respondents agreed or strongly agreed that each of these five principles helped 
them be more self-aware and self-disciplined (see Figure 8). Once again, respondents noted 
that the skills they developed through ACE had a strong influence on both their work and 
personal lives, with between one-fourth to one-third strongly agreeing with each statement as 
it related to “on the job,” and nearly one-fifth to one-third strongly agreeing with each as it 
related to their relationships. The aspect of self-awareness and self-discipline that was cited as 
having the strongest influence on both work (34%) and personal life (30%) was “to pause in 
difficult situations without immediately reacting,” with nearly one-third of respondents strongly 
agreeing. 


Figure 8. Influence of Self-Awareness and Self-Discipline Lessons on Work and 
Personal Life 


What | learned in the program helped me... 
m Strongly Agree © Agree m Disagree m Strongly Disagree @ Does Not Apply 


..to pause in difficult situations without immediately reacting. 
On the job 
In relationships 


..to understand when I’m in bioreaction. 
On the job 
In relationships 


...to realize |am personally capable of dealing with stress. 
On the job 
In relationships 


..to judge people less. 


On the job 
In relationships 


..to focus and concentrate. 


On the job 
In relationships 


0% 20% 40% 60% 80% 100% 


Academy for College Excellence: Mixed-Methods Analysis of Long-Term Outcomes 
The RP Group | March 2018 | Page 28 


Skill 3: Communication with Others 


Collaborative leadership skills, including communication skills, including listening to others, 
managing stressful conversations, and developing thoughtful, calm responses, play a chief role 
in facilitating participant growth on the job and in personal relationships. Based on participants 
replies to open-ended questions, the RP Group gleaned that this skill broke down the 
communication barrier that cripples some relationships—whether work or personal. One 
participant described, “It is apparent people can’t get out of the cycle of waste. It’s been tough 
to see it firsthand. [Better communication skills have] helped me reconnect with my family 
following my parents’ divorce.” 


, 


The RP Group selected the following five principles to investigate how the program helped 
students communicate: 


1. Understanding how to remain calm, stay positive, and manage emotions in the face 
of conflict and disagreement, and to consider ways to compromise to come to a 
desired solution or agreement (also termed in the ACE program as the “cycle of 
value/cycle of waste”) 

2. Becoming aware of emotions in self and others 

Listening for and understanding the needs, purpose, and concerns of others 

4. Understanding conversation meter (e.g., pretense, sincerity, accuracy, and 
authenticity) 

5. Ability to see the style of other people and adjusting my communication style 
appropriately 


w 
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The majority of respondents agreed or strongly agreed that the program helped them 
communicate with others in each of the ways listed above (see Figure 9). Additionally, 
respondents noted an evenly-balanced level of influence on work and personal life. 


Figure 9. Influence of Communication with Others Lessons on Work and Personal 
Life 
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Skill 4: Understanding Working Styles 


Understanding how individual styles influence team participation, which comprises ACE’s 
working styles skill, was underscored by participants as critical to job performance and personal 
relationships. This skill facilitated respondents’ comfort with and respect for diversity, increased 
their capacity to take another’s perspective, and informed their interactions with work 
colleagues, family, and friends. 


The RP Group selected the following three principles to examine how the program helped 
students understand learning and working styles: 


1. Understanding learning and working styles 
2. Identifying others’ learning and working styles 
3. Using learning and working styles in relationships and teams 


In particular, respondents agreed most strongly (29%) that “understanding my learning/working 
styles” impacted both their work (38%) and personal lives (29%) (See Figure 10). 


Figure 10. Influence of Working Style Lessons on Work and Personal Life 


What | learned in the program helped me... 
mStrongly Agree “Agree Disagree Strongly Disagree m Does Not Apply 
..to understand my learning/working style. 
On the job 


In relationships 


..to identify the learning/working styles of others. 


On the job 


In relationships 
..to use learning/working styles in relationships and teams. 


On the job 


In relationships 


..to use spider diagram charts to create teams. 


On the job 


In relationships 
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Skill 5: Social Justice Research 


The Social Justice Research Course was designed to advance research, presentation, and 
teamwork skills by assigning students an activity that requires they conduct primary research 
on a topic related to social justice in their community. 


The RP Group selected the following four principles to examine how the program helped 
students understand learning and working styles: 


1. Performing research, including development of a survey questionnaire 
2. Developing an action plan as part of a team presentation 

3. Creating and presenting using PowerPoint 

4. Contributing effectively to a team 
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The research and presentation skills honed and team work required as part of the social justice 
research course had a positive influence on respondents’ work lives particularly “how to 
contribute effectively to a team” to which the large majority (93%) agreed or strongly agreed 
(see Figure 11). 


Figure 11. Influence of Social Justice Research Course Lessons on Work and 
Personal Life 


What | learned in the program about... 
@ Strongly Agree m Agree m Disagree @ Strongly Disagree = Does Not Apply 
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.. how to contribute effectively to a team. 
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Conclusions 


Reflections on ACE’s Impact on Academic and Wage Outcomes 


Based on the analyses conducted for this study, there does appear to be positive academic and 
wage outcomes for students participating in ACE, with each ACE cohort design having varying 
impacts on outcomes. 


Enhancement of CTE Programs 


The positive impacts of ACE were found for students enrolled in a CTE program — nursing, 
where the ACE Foundation course was embedded in its curriculum. When compared to 
previous nursing student cohorts without the ACE Foundation course, a higher percentage of 
ACE nursing students completed an award within three and six years. Moreover, compared to a 
matched control group enrolled in the same timeframe, ACE nursing students earned 
significantly higher wages over time compared to their matched control group, than Pre-ACE 
nursing students and their matched control group. 


ACE as a Model for Closing Achievement and Wage Gaps 


Results from this study examined the impact of ACE participation for students who attempted 
the courses in various ACE cohort designs and for students who completed all the courses in the 
ACE cohort designs and found that the greatest positive academic outcomes were found for 
students who completed all the courses; especially in micro-level Group 7 (the group with the 
most fidelity to the ACE model). Students who completed the micro-level Group 7 were, in most 
cases, at least two times more likely than the control group to earn an award, complete a 
transfer-level English or math course, and earn 30 units within three years of enrolling in ACE. 


For wages, it was found that most students who completed the ACE cohort design that included 
accelerated English had higher wage gain increases over time compared to control group and 
the other ACE cohort designs (with the exception of the non-CTE Foundation course —which 
appeared to include outliers based on the variance found in the wages reported). 


Overall, although the absolute academic achievement and wages for ACE participants were not 
greater than those of the matched control group, the fact that the academic achievement was 
equal over time, was itself an achievement — especially given the known concentration of risk 
factors in the ACE group. Moreover, while absolute wages were not higher, largely due to the 
ACE students starting with lower wages, wage gains were twice as high for ACE students than 
for the matched peer set. ACE participation also appears to have the potential to close equity 
wage gaps. Wage comparisons for males of color and underrepresented minority (URM) 
student populations participating in ACE and their matched peers revealed lower wages overall 
for the ACE participants; but by the sixth year, wage gaps closed for the URM students who 
participated in ACE, and wages among males of color who participated in ACE exceeded those 
of their matched peers. 
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Reflections on ACE’s Impact on Personal Success 


The ACE program delivered on its mission—to propel student success in school, work, and life— 
in two central ways. First, the program supports students’ academic and social integration in 
higher education. Second, the ACE model delivers psychosocial supports for underserved and 
underrepresented college students, many of whom have faced social, familial, and economic 
hardships that have negatively affected their educational and career trajectories. Barillo-Sotillo, 
Miller, Nagasaka, and Arguelles (2009) argue these two achievements—strengthening academic 
and social integration, and providing psychosocial supports—are critical to low-income and 
minority students succeeding in community college. The following sections provide deeper 
insights related to how these academic, social, and psychosocial advances in the ACE program 
impact participants’ postsecondary success, ability to work in teams, and the development of 
noncognitive skills. 


Postsecondary Success for Underrepresented Students 


As cited in Barrio-Sotillo, Miller, Nagasaka, and Arguelles “academic integration involves the 
development of a strong affiliation with the college academic environment both inside and 
outside of the classroom through interactions with faculty, staff, and peers in an academic 
nature” (2009, p. 266). For a high- or at-risk population, which describes all of the survey 
respondents for whom risk data were available, that has likely encountered crime, 
psychological stress, gang influence, drug and alcohol exposure, homelessness, and other 
obstacles, this accomplishment cannot be overstated. Many of these students may have been 
hesitant to see themselves as college bound, but the ACE model provided a supportive 
classroom with which to view their potential. The development of this view of oneself, has 
financial and career advantages. Although the academic and wage outcomes for ACE students 
were lower overall, it is remarkable that over the long-term, their academic achievements 
equaled, or in some cases even exceeded those of other students who began the academic 
“race” much closer to the finish line and saw more growth in their earnings over a six-year 
period than did their peers from the control group. 


The Power of Teamwork 


Social integration in the form of team work was key to the student experience in ACE. Barrio- 
Sotillo, Miller, Nagasaka, and Arguelles (2009) argue that social integration—largely faculty and 
peer driven—reinforces persistence and retention among underserved populations in the 
community college. The ACE curriculum was so powerful that up to 11 years later, some 
students could recall core memories related to working effectively in a team. Participants 
relayed stories about working styles, developing a sense of trust and openness with classmates, 
and receiving the support of their peers. These memories helped researchers form an 
understanding of the ways in which ACE respondents valued the social dynamics of the 
program and applied the curricula to their everyday lives (e.g., being collegial, addressing new 
challenges, understanding others). 
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The Value of Strong Noncognitive Skills 


The ACE program offered psychosocial supports that advanced students’ noncognitive or 
affective skills. Former students’ survey responses highlighted how the noncognitive skills— 
pausing in difficult situations, being confident in their ability and power to shape their lives, 
controlling their emotions—helped them in all aspects of their work and personal lives. 
Cognitive skills such as understanding working styles, setting priorities, and adjusting their 
communication style as dictated by the conversation or situation were also important, but 
respondents most often offered examples of how the affective elements of the personal 
success Skills had positively influenced them. In the ACE classroom, the advancement of these 
critical skills materialized through a curricular framework and pedagogical approaches that 
embraced adult learning principles, faculty-led real-world scenarios, and the creation of a safe 
space for students to share deeply personal stories. 


Limitations of the Findings 


Qualitative Study 


Limitations beyond the control of the RP Group constrained portions of the qualitative study. 
First, together a limited budget and time constraints required the selection of a telephone 
survey that was to the point, including many structured, closed-ended questions. The 
advantage was that this instrument allowed the RP Group to swiftly gather responses from a 
large sample population to maximize perspectives, address the primary research questions, and 
stay on budget. The disadvantage was that the telephone survey did not exceed 15 minutes and 
there were no opportunities to ask follow-up questions. Second, the RP Group was unsuccessful 
in recruiting ACE participants for focus group interviews to augment the telephone survey. In an 
attempt to gather more nuanced insights and details on the program’s perceived influence on a 
randomly-selected sample, the RP Group aimed to conduct multiple hour-long focus groups 
with 120 telephone survey respondents. However, recruitment invitations from both the RP 
Group and ACE Founder Diego Navarro yielded only two completed one-on-one interviews. 
Since these interviews did not reflect a representative sample of ACE participants, the results of 
these conversations have not been included in the research findings presented here. Third, the 
study lacked access to employer perspectives. Originally, the design of this mixed-methods 
study included surveying a subset of current employers of former ACE participants. However, 
fewer than 10% of survey respondents agreed to share their employers’ contact information, 
and ultimately the majority of the employers who were contacted declined to be interviewed or 
did not respond to the researchers’ invitations. 


Quantitative Study 


While the findings of this research reveal important associations between participation in ACE 
and academic, employment, and personal success, it is critical to view them with an 
understanding of the limitations inherent to an analysis of this kind. First, the ACE model 
targets high-risk student populations, and although the study was able to identify a statistically- 
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matched comparison group; the matching was limited to information that was collected and 
recorded at each college such as basic demographics (e.g., ethnicity, gender) and enrollment 
history; the data for risk characteristics was missing for all non-ACE students, making it 
impossible to determine if the matched peers also shared the same high-risk factors. In 
addition, even though the models from the propensity score analyses performed very well, with 
all overall models showing non-significant results after matching, the analyses can only account 
for the observed differences and does not account for other confounding factors. That is, this 
adjustment does not entirely eliminate the problem of selection bias, but increases our 
confidence that the results are not entirely due to preexisting differences between the 
treatment (ACE) and control groups with regards to the selected variables. 


Second, EDD only provides matched wage data aggregated with a minimum of five records, 
therefore, the RP Group could not conduct any unitary record analyses with the wage data. 
Third, wage data can be only be provided for students with valid social security numbers. 
Fourth, EDD wage data only provides wages for employers who pay into the unemployment 
insurance funds and therefore, do not contain wages for workers such as those who are 
migrant/seasonal workers, federal employees (including military), or self-employed. Given this 
limitation, not all students identified in this study will have wage information, and the inability 
to access unitary data meant the RP Group could not determine whether the wage information 
is missing for the reasons identified or if a student was unemployed. 


Future Research 


The current study found that the addition of an ACE Foundation course in an allied health 
science program such as nursing, appeared to have the most impact on completion and wage 
gains, perhaps indicating a synergistic effect between a sought-for credential with underlying 
training that bolsters performance while in school and on the job. However, it was noted that 
state unemployment rates varied greatly across the years this study covered and the 
unemployment rates were highest between 2009 and 2011, affecting the employment 
opportunities for students trying to enter the labor market during that time. As a result, it is 
recommended additional research be conducted on multiple CTE pathways with embedded 
ACE components to further explore and validate the potential impact of ACE on wage outcomes 
for CTE students. 
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Appendix A: ACE Cohort Variations 


Table A1. ACE Course Configurations for Meta- and Micro-Level Groupings: 


Percentage of Attempters and Completers 


ACE 
Grouping ACE Cohort Design and Included Courses 


Total/Percent 


Total/Percent 


ID of Attempters? of Completers? 
CTE TRACK 
Meta-Group 1: Career/Technical Education 236 (7.9%) 236 (7.9%) 


1CTE e Foundation (nursing students) 


NON-CTE TRACK 


Meta-Group 2: Accelerated English/ACE Model 


e Foundation 

e Team self-management 
e Social justice 

e Accelerated English 


e Foundation 

e Team self-management 
5 e Social justice 

e Accelerated English 

e Movement 


e Foundation 
e Team self-management 
6 e Social justice 
e Accelerated English 
e Career Development 
e Foundation 
e Team self-management 
e Social justice 
e Accelerated English 
e Movement 
e Career Development 


236 (7.9%) 


1,549 (51.9%) 


518 (17.4%) 


501 (16.8%) 


234 (7.8%) 


296 (9.9%) 


236 (7.9%) 


819 (27.6%) 


313 (11.4%) 


246 (8.9%) 


98 (3.6%) 


162 (5.9%) 
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ACE 
Grouping ACE Cohort Design and Included Courses 


Total/Percent 


Total/Percent 


ID of Attempters of Completers 
Meta-Group 3: Non-Accelerated English/Alternative to 443 (14.9%) 541 (18.3%) 


the ACE Model 


e Foundation 
e Team self-management 
e Optional Social Justice 


10 e Optional Movement 
e Optional Career Development (at Cabrillo 
College) 


e Un-Accelerated English 
e Foundation 
e Optional Team self-management 
e Optional Social Justice 
11 e Optional Movement 
e Optional Career Development 
e Un-Accelerated English (at Cabrillo 
College) 


Meta-Group 4: Limited Resources 


e Foundation 
e Team self-management 
e Foundation 
3 e Team self-management 
e Social Justice 
e Foundation 
e Team self-management 
e Optional Social Justice 


2 


: e Accelerated English 
e Optional Movement 
e Optional Career Development 
e Foundation 
e Optional Team self-management 
9 e Optional Social Justice 
e Accelerated English 
e Optional Movement 
e Optional Career Development 
e Foundation, Team self-management 
e Optional Social Justice 
12 e Accelerated English 


e Optional Movement 
e Optional Career Development (at 
Hartnell) 


376 (12.6%) 


67 (2.2%) 


596 (20.0%) 


147 (4.9%) 


114 (3.8%) 


142 (4.8%) 


171 (5.7%) 


14 (0.5%) 


267 (8.9%) 


274 (9.2%) 


720 (24.3%) 


138 (4.7%) 


171 (5.7%) 


94 (3.2%) 


210 (7.1%) 


66 (2.2%) 


Academy for College Excellence: Mixed-Methods Analysis of Long-Term Outcomes 


The RP Group | March 2018 | Page 41 


ACE 
Grouping ACE Cohort Design and Included Courses 
ID 
e Foundation 
e Optional Team self-management 
e Optional Social Justice 
13 e Accelerated English 
e Optional Movement 
e Optional Career Development (at 
Hartnell) 


Meta-Group 5: Foundation Course Only 


1 —Non- 


CTE e Foundation 


Total/Percent 
of Attempters 


8 (0.3%) 


158 (5.3%) 


158 (5.3%) 


Total/Percent 
of Completers 


41 (1.4%) 


647 (21.8%) 


647 (21.8%) 


Notes. ?Students who attempted all the courses in the model variation. ’ Students who completed all the courses 
in the model variation. Attempters and completers are not mutually exclusive groups and so depending on 
students’ actual performances in the courses, the number of completers in any given group may greater than the 
number of attempters. For example, a student may be flagged as an attempter in Meta-Group 2 if the student 
attempted a Foundation, team self-management, social justice research, and accelerated English course. However, 
if the student passes all the courses except the accelerated English course, the student will be flagged as completer 


in Meta-Group 3. 
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Appendix B: Propensity Score Analyses 


Table B1. List of Variables Used for Propensity Score Matching Analyses 


Variable 


Description 


Levels 


Age 


Student’s age in ACE term 


Continuous 


CA Resident 


Student is a California resident 


Yes, No 


Census: Below High School 
Achievement (%) 


Percent of residents with 
highest educational level 
below high school. Based on 
student ZIP code and five-year 
averages from 2015 American 
Community Survey (ACS) 


Continuous 


Census: Poverty (%) 


Percent of residents living in 
poverty. Based on student ZIP 
code and five-year averages 
from 2015 ACS survey 


Continuous 


Census: Unemployment (%) 


Percent of residents who are 
unemployed. Based on 
student ZIP code and five-year 
averages from 2015 ACS 
survey 


Continuous 


Census: White/Caucasian (%) 


Percent of White or Caucasian 
residents. Based on student 
ZIP code and five-year 
averages from 2015 ACS 
survey 


Continuous 


Educational Goal 


Student’s self-reported 
educational goal 


Certificate/AA, Transfer/BA, 
undecided, or missing 


Educational Level 


Student level of educational 
achievement in ACE term 


Below high school, GED, high 
school, higher education, or 
missing 


English Course Level in Term 


A student’s English course 
level in ACE term 


Two+ levels below transfer, 
one level below transfer, or 
transfer level 


Math Course Level in Term 


A student’s math course level 
in ACE term 


Two+ levels below transfer, 
one level below transfer, or 
transfer level 


Ethnicity A student’s self-reported Asian, African American/Black, 
ethnicity Hispanic, White, or Other 
Exempt A student is exempt from Yes, No 


matriculation (e.g., 
orientation, education plan, 
and assessment testing) 


Academy for College Excellence: Mixed-Methods Analysis of Long-Term Outcomes 


The RP Group | March 2018 | Page 43 


Variable Description Levels 

Financial Aid Amount Student financial aid amount Continuous 
received 

Financial Aid Received Student received financial aid Yes, No 
for the academic year 

Foster Youth Student self-reported as Yes, No 
former or current foster youth 

Gender A student’s self-reported Male, Female 


gender 


Prior English Achievement 


A student’s completed level of 


English prior to ACE term 


Two+ levels below transfer, 
one level below transfer, or 
transfer level 


Prior Math Achievement 


A student’s completed math 
level prior to ACE term 


Two+ levels below transfer, 
one level below transfer, or 
transfer level 


Prior GPA Cumulative GPA prior to term Continuous 
GPA=O were screened out 

Prior Main Terms Attended Number of main terms (fall or |= Continuous 
spring) attended prior to term 

Prior Terms Attended Number of terms attended Continuous 
prior to term 

Prior Units Earned Number of units earned prior Continuous 


to term 
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Figure B1. Dot Graph Before and After Matching — Berkeley City College 


0 
Standardized % bias across covariates 
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Figure B2. Dot Graph Before and After Matching — Cabrillo College 
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Figure B3. Dot Graph Before and After Matching — Hartnell College 
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Figure B4. Dot Graph Before and After Matching — Pre-ACE Nursing (Hartnell) 


® Unmatched 
x Matched 


Standardized % bias across covariates 


Notes. When forming a comparison group for the ACE nursing students, we first sought to include nursing students 
who had not been required to participate in ACE--these would be students who began their enrollment prior to 
2009. However, this comparison was problematic in at least two respects. First, labor market outcomes could be 
very different between students who started before 2009 and those starting after 2009, owing to changes in the 
economy as the market moved from recession to recovery. Moreover, matching between pre-ACE and ACE nursing 
students did not create two comparable groups. As a result, we decided to match pre-ACE nursing students with 
non-nursing students enrolled during the same time, and ACE nursing students with non-nursing students enrolled 
at the same time, and then examine how each nursing cohort compared with its contemporaneous non-nursing 
counterparts. 
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Figure B5. Dot Graph Before and After Matching — ACE Nursing (Hartnell) 
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Appendix C: Academic Outcomes 


Table C1. Comparison of Academic Outcome Attainment 


Outcome ACE Participants Control Group Difference 
Number of Percentage of Number of Percentage 
Students Students Students of Students 

Completion of a Degree or Certificate 
Within 3 years 68 3.0% 106 4.8% -1.78* 
Within 6 years 95 9.0% 138 12.1% -3.12 
Transfer to a Four-Year Institution 
Within 3 years 28 1.3% 89 4.2% -2.85*** 
Within 6 years 57 6.4% 106 9.5% -3.14* 
Successful Completion of Transfer-Level English 
Within 3 years 470 21.3% 549 26.0% -4.64*** 
Within 6 years 200 22.7% 313 28.4% -5.7* 
Successful Completion of Transfer-Level Math 
Within 3 years 151 6.7% 168 7.7% -0.94 
Within 6 years 94 10.4% 135 11.7% -1.32 
Transfer Prepared 
Within 3 years 57 2.5% 99 4.5% -1.91*** 
Within 6 years 64 7.0% 106 9.1% -2.05 
Completion of 30 units 
Within 3 years 330 14.8% 421 19.5% -4.68*** 
Within 6 years 195 21.7% 288 25.4% -3.64 


Notes. * = ps.05, **ps.01, ***p <.001. The ACE students included in this analysis are only those categorized as 
“non-CTE”—in other words, participants who went on to pursue an academic rather than career/technical 
education pathway, such as nursing. Only students enrolled in academic years where at least three- and six-year 
rates could be calculated were included in the analysis. The cohort for three-year rates were students enrolled 
between fall 2003 and spring 2013 who achieved an outcome within three years of the date on which they joined 
an ACE program; and the cohort for six-year rates were students enrolled between fall 2003 and spring 2010 who 


achieved an outcome within six years of the date on which they joined an ACE program. 
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Table C2. Percentage Comparisons Attained Academic Outcomes by Under- 


represented Minority (URM) Non-CTE ACE Participants and URM Controls 


Outcome Non-CTE ACE Control Group 
Comparison 
Total in eee Total in students who Difference 
Sample Sample attained 
outcome Bitceine 
N N (%) N N (%) % 
Completion of a degree or 
certificate 
Within 3 years 1,576 36 (2.28%) 1,646 72 (4.37%) 2.09%*** 
Within 6 years 616 59 (9.58%) 836 91(10.89%) 1.31% 
Transfer to a four-year 
institution 
Within 3 years 1,560 19 (1.22%) 1,597 69 (4.32%) 3.1%*** 
Within 6 years 614 41 (6.68%) 814 71 (8.72%) 2.04% 
Successful completion of 
transfer-level English 
Within 3 years 1,555 327 (21.03%) 1,580 425 (26.90%) 5.87%** 
Within 6 years 605 133 (21.98%) 803 235 (29.27%) 7.29%*** 
Successful completion of 
transfer-level math 
Within 3 years 1,575 114 (7.24%) 1,623 122 (7.52%) 0.28% 
Within 6 years 618 66 (10.68%) 834 98 (11.75%) 1.07% 
Transfer Prepared 
Within 3 years 1,577 49 (3.11%) 1,652 76 (4.60%) 1.49%* 
Within 6 years 619 49 (7.92%) 842 75 (8.91%) 0.99% 
Completion of 30 units 
Within 3 years 1,570 237 (15.1%) 1,610  309(19.19%) 4.09%* 
Within 6 years 613 129 (21.04%) 822 214 (26.03%) 4.99%* 


Notes: * = ps.05, **ps.01, ***p <.001. Only students enrolled in academic years where at least three- and six-year 
rates could be calculated were included in the analysis. The cohort for three-year rates were students enrolled 
between fall 2003 and spring 2013 who achieved an outcome within three years of the date on which they joined 
an ACE program; and the cohort for six-year rates were students enrolled between fall 2003 and spring 2010 who 
achieved an outcome within six years of the date on which they joined an ACE program. 
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Table C3. Logistics Regression Results for ACE Meta-Level Groupings Using Control 


Group as Reference Group: Three- and Six-Year Rates (Enrolled) 


3 years 6 years 
OR SE z Cl OR SE z Cl 

Awards 

Accelerated English 0.32*** -0.08 -4.56 [0.19,0.52] 0.85 -0.19 -0.7 [0.55,1.33] 
Un-Accelerated 0.84 -0.22 -0.69 [0.50,1.39] 0.85 -0.16 -0.85 [0.59,1.23] 
English 

Limited Resources 0.54* -0.16 -2.07 [0.30,0.97] 0.68 -0.17 -1.55 [0.41,1.11] 
Foundation 122) -0.46 0.52 [0.58,2.55] 1.16 -0.73 0.23 [0.34,3.95] 
English 

Accelerated English 0.85* -0.07 -1.96 [0.72,1.00] 0.94 -0.15 -0.41 [0.69,1.28] 
Un-Accelerated 0.49*** -0.07 -4.94 [0.37,0.65] 0.57*** -0.08 -3.88 [0.43,0.76] 
English 

Limited Resources 0.97 -0.11 -0.28 [0.77,1.22] 0.89 -0.15 -0.72 [0.64,1.23] 
Foundation 0.60* -0.14 -2.14 [0.37,0.96] 0.32 -0.24 -1.52 [0.07,1.39] 
Math 

Accelerated English 0.65** -0.1 -2.81 [0.48,0.88] 1.1 -0.23 0.46 [0.73,1.66] 
Un-Accelerated O.S7* -0.14 -2.3  [0.36,0.92] 0.62* -0.13 -2.26 [0.42,0.94] 
English 

Limited Resources 1.54**  -0.25 2.64 [1.12,2.13] 1.01 -0.22 0.05 [0.66,1.56] 
Foundation 1.56 -0.45 1.55 [0.89,2.73] 0.83 -0.62 -0.25 [0.19,3.62] 


Academy for College Excellence: Mixed-Methods Analysis of Long-Term Outcomes 
The RP Group | March 2018 | Page 52 


3 years 6 years 

OR SE z Cl OR SE z Cl 
Transfer 
Accelerated English O.22*** 0:07 -4.68 [0.12,0.42] 0.81 -0.21 -0.82 [0.49,1.35] 
Un-Accelerated 0.16** -0.1 -3.08 [0.05,0.52] 0.51** -0.13 -2.68 [0.31,0.84] 
English 
Limited Resources 0.50* -0.17 -2.03 [0.26,0.98] 0.58 -0.18 -1.8 [0.32,1.05] 
Foundation 0.55 -0.33 -1.01 [0.17,1.76] 2.56 -1.47 1.65 [0.84,7.87] 
Transfer Prepared 
Accelerated English 0.27*** -0.08 -4.65 [0.16,0.47] 1.01 -0.24 0.02 [0.63,1.61] 
Un-Accelerated 0.20** -0.1 -3.18 [0.07,0.54] 0.54* -0.13 -2.47 [0.34,0.88] 
English 
Limited Resources 1.22 -0.28 0.86 [0.78,1.89] 0.77 -0.21 -0.98 [0.45,1.31] 
Foundation 2.09* -0.67 2.31. ([1,12,3:91] 1-65 -1.04 0.79 [0.48,5.68] 
30 Units Completion 
Accelerated English 0.66*** -0.07 -4.12 [0.54,0.80] 0.89 -0.15 -0.67 [0.65,1.24] 
Un-Accelerated 0.60*** -0.09 -3.31 [0.44,0.81] 0.69* -0.1 -2.55 [0.52,0.92] 
English 
Limited Resources 1.03 -0.13 0.24 [0.80,1.32] 1.03 -0.17 0.16 [0.74,1.42] 
Foundation 0.87 -0.2 -0.61 [0.55,1.37] 0.3 -0.22 -1.61 [0.07,1.30] 


Note. *ps.05, **ps.01, ***p <.001. OR = Odds Ratio. SE = Standard Error. Z = Z-score. CI = Confidence Interval. 
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Table C4.Logistics Regression Results for ACE Meta-Level Groupings Using Control 
Group as Reference Group: Three- and Six-Year Rates (Completed) 


3 years 6 years 
OR SE z Cl OR SE z Cl 

Awards 

Accelerated English 0.58* -0.15  -2.12 [0.35,0.96] 1.26 -0.32 0.91 [0.77,2.07] 
Un-Accelerated 0.74 -0.19 -1.17  [0.44,1.23] 1.02 -0.19 0.13 [0.71,1.48] 
English 

Limited Resources 0.49* -0.14 -2.45 [0.28,0.87] 0.9 -0.21 -0.45 [0.57,1.43] 
Foundation 0.29*** -0.11 -3.31 [0.14,0.61] 0.15*** -0.08 -3.72 [0.05,0.41] 
English 

Accelerated English 1.69*** -0.16 5.44 [1.40,2.05] 1.58* -0.29 2.46 [1.10,2.27] 
Un-Accelerated 0.48*** -0.07. -5.28 [0.37,0.63] 0.70* -0.1 -2.41 [0.52,0.94] 
English 

Limited Resources 0.98 -0.11 -0.2 [0.797121 1-12 -0.18 0.72 [0.82,1.55] 
Foundation 0.17*** -0.03 -9.05 [0.11,0.25] 0.09*** -0.04 -6.03 [0.04,0.20] 
Math 

Accelerated English 1.15 -0.19 0.88 [0.84,1.59] 1.58 -0.38 1.93 [0.99,2.53] 
Un-Accelerated 0.51** -0.12 -2.82 [0.31,0.81] 0.75 -0.16 -1.41 [0.49,1.12] 
English 

Limited Resources 1.34 -0.22 1.82 [0.98,1.84] 1.33 -0.28 1.37  [0.88,2.01] 
Foundation 0.44*** -0.11 -3.3 [0.27,0.72] 0.15*** -0.08 -3.65 [0.06,0.42] 
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3 years 6 years 

OR SE z Cl OR SE z Cl 
Transfer 
Accelerated English 0.35** -0.12 -3 [0.17,0.69] 1.15 -0.34 0.49 [0.65,2.05] 
Un-Accelerated 0.19** -0.1 -3.2 [0.07,0.53] 0.60* -0.15 -1.99 [0.37,0.99] 
English 
Limited Resources 0.42* -0.14 -2.56 [0.22,0.82] 0.77 -0.21 -0.94 [0.44,1.33] 
Foundation 0.18*** -0.09 -3.35 [0.07,0.49] 0.25** -0.12 -2.95 [0.10,0.63] 
Transfer Prepared 
Accelerated English 0.45** -0.13 -2.66 [0.25,0.81] 1.61 -0.42 1.81 [0.96,2.68] 
Un-Accelerated 0.17*** -0.09 -3.41 [0.06,0.47] 0.65 -0.16 -1.77 [0.40,1.05] 
English 
Limited Resources 1.03 -0.23 0.11 [0.66,1.60] 0.94 -0.24 -0.24 [0.56,1.56] 
Foundation 0.57 0.17. -1.94 [0.32,1.01] 0.20** -0.1 -3.11 [0.07,0.55] 
30 Units Completion 
Accelerated English 1.23 -0.14 1.87 [0/99;1.52] 1.38 -0.27. 1.69 [0.95,2.02] 
Un-Accelerated 0.55*** -0.08 -3.99 [0.40,0.73] 0.85 -0.12 -1.13 [0.63,1.13] 
English 
Limited Resources 0.95 -0.12 -0.45 [0.75,1.20] 1.24 -0.2 1.34 [0.90,1.71] 
Foundation 0.26*** -0.05 -6.98 [0.18,0.38] 0.14*** -0.05 -5.63 [0.07,0.28] 


Note. *ps.05, **ps.01, ***p <.001. OR = Odds Ratio. SE = Standard Error. Z = Z-score. CI = Confidence Interval. 
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Table C5.Logistics Regression Results for ACE Micro-Level Groupings Using Control 
Group as Reference Group: Three- and Six-Year Rates (Attempted) 


3 years 6 years 
OR SE z Cl OR SE z Cl 

Awards 

1 1.22 -0.46 0.52 [0.58,2.55] 1.16 -0.73 0.23 [0.34,3.95] 
2 0.43 -0.25 -1.44 [0.13,1.36] 0.5 -0.19 -1.82 [0.24,1.06] 
3 0.18 -0.18 -1.71 [0.02,1.29] 0.73 -0.77.  -0.3 [0.09,5.76] 
4 O.37** -0.14 -2.71 [0.18,0.76] 0.51 -0.31 -1.11 [0.16,1.67] 
5 0.11** -0.08 -3.14 [0.03,0.43] 0.63 -0.26 -1.13 [0.29,1.40] 
6 0.22 -0.22  -1.5 [0.03,1.59] 0.59 -0.43 -0.72 [0.14,2.50] 
7 0.55 -0.2 -1.61 [0.26,1.14] 1.38 -0.42 1.07 [0.76,2.52] 
8 0.77 -0.36 -0.55 [0.31,1.93] 0.86 -0.31 -0.43 [0.42,1.74] 
9 0.69 -0.41 -0.63 [0.21,2.21] 0.67 -0.7 -0.39 [0.09,5.19] 
10 0.99 -0.26 -0.03 [0.59,1.65] 0.96 -0.18 -0.23 [0.66,1.40] 
14 1 (.) [1.00,1.00] 0.35 “0,21. <1,74 [0411.14] 
12 1.62 -1.7 0.46 [0.21,12.61] 3.66 -4.49 1.06 [0.33,40.61] 
English 

1 0.60* -0.14 -2.14 [0.37,0.96] 0.32 -0.24 -1.52 [0.07,1.39] 
2 0.49** -0.12  -2.86 [0.30,0.80] 0.43** -0.12 -3.14 [0.25,0.73] 
3 0.47** -0.13 -2.71 [0.27,0.81] 0.57 -0.44 -0.73 [0.12,2.63] 
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3 years 6 years 
OR SE z Cl OR SE z Cl 

4 0.98 -0.12 -0.19 [0.77,1.24] 0.65 -0.25 -1.12 [0.31,1.38] 
5 0.54*** -0.08 -4.09 [0.40,0.72] 0.53* -0.15 -2.18 [0.30,0.94] 
6 0.7 -0.19 -1.29 [0.41,1.20] 1.75 -0.69 1.41 [0.80,3.81] 
7 1.15 -0.16 1.01 [0.88,1.51] 1.41 -0.33 1.46 [0.89,2.23] 
8 1.60* -0.3 2.47 [1.10,2.32] 1.91** -0.44 2.8 [1.21,3.00] 
9 TS kes -0.43 2.89 e232 95 |e -0.78 0.39 [0.38,4.25] 
10 0.52*** -0.08 -4.2 [0.39,0.71] 0.61** -0.09 -3.18 [0.45,0.83] 
11 O2oFe -0.12  -2.88 [0.12,0.67] 0.36** -0.14 -2.69 [0.17,0.76] 
12 0.87 -0.57  -0.22 [0.24,3.16] 1 (.) [1.00,1.00] 
Math 

1 1.56 -0.45 1.55 [0.89,2.73] 0.83 -0.62 -0.25 [0.19,3.62] 
2 0.63 -0.25 -1.17  [0.29,1.37] 0.58 -0.21 -1.5 [0.29,1.18] 
3 0.22* -0.16 -2.1 [0.05,0.90] 0.75 -0.79 -0.28 [0.09,5.88] 
4 0.46**  -0.12 -2.92 [0.27,0.77] 0.91 -0.44 -0.19 [0.35,2.35] 
5 0.40**  -0.12 -3.01 [0.22,0.73] 0.85 -0.31 -0.44 [0.42,1.74] 
6 1.01 -0.4 0.02 [0.46,2.22] 1.7 -0.86 1.05 [0.63,4.56] 
7 1.17 -0.26 0.7 [0.76,1.80] 1.3 -0.41 0.82 [0.70,2.40] 
8 2.05** 053.0274 (0.235340) 87> -0.53 2.18 [1.07,3.27] 
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3 years 6 years 
OR SE z Cl OR SE z Cl 

9 4.74*** -1.18 6.24 [2.91,7.74] 0.68 -0.71 -0.37  [0.09,5.30] 
10 0.57* -0.15 -2.14 [0.34,0.95] 0.57* -0.13 -2.41 [0.36,0.90] 
11 0.57 -0.34 -0.95 [0.18,1.83] 0.9 -0.37  -0.25 [0.40,2.02] 
12 hale) 168 -1 [0.48,9.89] 1 (.) [1.00,1.00] 
Transfer 

1 0.55 -0.33 -1.01 [0.17,1.76] 2.56 -1.47 1.65 [0.84,7.87] 
2 0.52 -0.31 -1.1 [0.16,1.67] 0.24* -0.15 -2.37 [0.08,0.78] 
3 0.45 -0.32 -1.11 [0.11,1.84] 1.2 -1.28 0.17 [0.15,9.70] 
4 0.22**  -0.11 -2.92 [0.08,0.61] 0.94 -0.5  -0.12 [0.33,2.67] 
5 0.13** -0.09 -2.87 [0.03,0.52] 0.58 -0.27. -1.16 [0.23,1.46] 
6 1 (.) [1.00,1.00] 1 (.) [1.00,1.00] 
7 0.41 -0.19 -1.92 [0.17,1.02] 1.28 -0.45 0.71 [0.64,2.56] 
8 0.55 -0.33 -0.99 [0.17,1.78] 1.15 -0.42 0.39 = [0.56,2.37] 
S) 0.55 -0.4 -0.83 [0.13,2.26] 1 (a) [1.00,1.00] 
10 0.19** -0.11 -2.8 [0.06,0.61] 0.52* -0.14 -2.45 [0.31,0.88] 
11 1 (.) [1.00,1.00] 0.47 -0.28 -1.25 [0.15,1.53] 
12 1 (.) [1.00,1.00] 1 (.) [1.00,1.00] 
Transfer 

Prepared 
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3 years 6 years 
OR SE z Cl OR SE z Cl 

1 2.09* -0.67 2.31 [1.12,3.91] 1.65 -1.04 0.79 [0.48,5.68] 
2 0.47 -0.28 -1.29 [0.15,1.49] 0.42 -0.19 -1.88 [0.17,1.04] 
3 0.2 -0.2 -1.62 [0.03,1.41] 1 (.) [1.00,1.00] 
4 O20"= -0.1 -3.16 [0.07,0.54] 0.45 -0.33 -1.1 [0.11,1.88] 
5 0.17** -0.1 -2.97 [0.05,0.55] 0.85 -0.35 -0.39 [0.38,1.90] 
6 0.24 -0.24 -1.42 [0.03,1.73] 1.23 -0.77. 0.34 = [0.37,4.17] 
7 0.52 -0.21 -1.64 [0.24,1.14] 1.41 -0.48 1.02 [0.73,2.74] 
8 1.38 -0.52 0.86 [0.66,2.91] 1.6 -0.52 1.44  [0.84,3.04] 
9 3.74*** -1.19 4.15 [2.00,6.96] 1 (.) [1.00,1.00] 
10 O.23** -0.12  -2.85 [0.08,0.63] 0.56* -0.15 -2.22 [0.33,0.93] 
11 1 (.) [1.00,1.00] 0.48 -0.29 -1.23  [0.15,1.55] 
12 1.77 -1.85 0.55 [0.23,13.76] 1 (.) [1.00,1.00] 
30 Units 

1 0.87 -0.2 -0.61 [0.55,1.37] 0.3 -0.22 -1.61 [0.07,1.30] 
2 0.75 -0.18 -1.18 [0.47,1.21] 0.8 -0.18 -0.97 [0.51,1.26] 
5} 0.20*** -0.09 -3.46 [0.08,0.50] 1.13 -0.77. 0.18 = [0.30,4.29] 
4 0.43*** -0.08 -4.79 [0.31,0.61] 0.54 -0.23 -1.48 [0.24,1.22] 
5 0.49*** -0.09 -4 [0.34,0.69] 0.52* -0.16 -2.11 [0.29,0.96] 
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3 years 6 years 
OR SE z Cl OR SE z Cl 
6 1.06 -0.29 0.23 [0.63,1.80] 2.07 -0.82 1.83 [0.95,4.52] 
7 1.17 -0.18 1.04 [0.87,1.58] 1.26 -0.31 0.96  [0.78,2.04] 
8 1.43 -0.3 1.73 [0.95,2.16] 1.45 -0.35 1.56 [0.91,2.33] 
9 2.68*** -0.6 4.37 [1.72,4.17] 1 -0.67 0.01 [0.27,3.74] 
10 0.66* -0.11 -2.57  [0.48,0.91] 0.75 -0.11 -1.91 [0.55,1.01] 
11 0.27* -0.14 -2.49 [0.10,0.76] 0.42* -0.16 -2.24 [0.20,0.90] 
12 0.35 -0.37 -0.99 [0.05,2.74] 1 (.) [1.00,1.00] 


Note. *ps.05, **ps.01, ***p <.001. OR = Odds Ratio. SE = Standard Error. Z = Z-score. CI = Confidence Interval. 
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Table C6. Logistics Regression Results for ACE Micro-Level Groupings Using 
Control Group as Reference Group: Three and Six Year Rates (Completed) 


3 year 6 year 

OR SE z Cl OR SE z Cl 
Awards 
1 0.29*** -0.11 -3.31 [0.14,0.61] 0.15*** -0.08 -3.72 [0.05,0.41] 
2 0.62 -0.32 -0.93 [0.22,1.71] 0.88 -0.31 -0.36 [0.45,1.74] 
3 0.28 -0.2 -1.78 [0.07,1.14] 0.61 -0.45  -0.67 [0.14,2.61] 
4 0.56 -0.22 -1.47 [0.26,1.21] 0.32 -0.33 -1.12 [0.04,2.37] 
5 0.23* -0.16 -2.06 [0.06,0.93] 1.02 -0.45 0.05 [0.43,2.44] 
6 0.46 -0.47 -0.76 [0.06,3.40] 1.05 -0.8 0.06 [0.24,4.65] 
7 1.04 -0.39 0.1 [0.50,2.17] 2.09* -0.71 2.17 [1.07,4.07] 
8 ses) -0.54 0.29 [0.46,2.88] 1.57 -0.59 1.19 [0.75,3.29] 
9 0.52 -0.31 -1.11 [0.16,1.66] 0.98 -0.74 -0.03 [0.22,4.31] 
10 1.47 -0.39 1.47 [0.88,2.47] 1.60* -0.32 2.32 [1.08,2.38] 
11 1 (.) [1.00,1.00] 0.27** -0.13 -2.81  [0.11,0.67] 
12 1 (.) [1.00,1.00] 0.24 -0.24 -142 [0.03,1.74] 
English 
1 0.17*** -0.03 -9.05 [0.11,0.25] 0.09*** -0.04 -6.03  [0.04,0.20] 
2 0.73 -0.17. -1.38 [0.46,1.14] 0.74 -0.19 -1.16 [0.44,1.23] 
3 0.45** -0.11 -3.15 [0.28,0.74] 0.33 -0.2 -1.79 [0.10,1.11] 
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3 year 6 year 
OR SE z Cl OR SE z Cl 

4 1.65*** -0.23 3.57 [1.25,2.17] 0.71 -0.36 -0.68 [0.26,1.92] 
5 1.21 -0.21 1.08 [0.86,1.70] 0.82 -0.28 -0.57 [0.42,1.60] 
6 1.96* -0.62 2.12 [1.05,3.66] 5.59** -3.04 3.17 [1.93,16.23] 
7 Ze 5 aod 5.17 [1.71,3.31] 2.45** -0.69 3.17 [1.41,4.26] 
8 3.09*** -0.67 5.17 [2.01,4.73] 4.15*** -1.24 4.76 [2.31,7.45] 
9 1.88** -0.37 3.19 [1.28,2.77] 2.23 -1.16 1.53 [0.80,6.19] 
10 0.84 -0.13 -1.11 [0.61,1.14] 1.04 -0.18 0.22 [0.75,1.45] 
11 0.17*** -0.05 -6.38 [0.10,0.29] 0.31*** -0.09 -4.16 [0.18,0.54] 
12 0.18*** -0.08 -3.72 [0.07,0.44] 0.36 -0.2 -1.88 [0.13,1.04] 
Math 

1 0.44*** -0.11 -3.3  [0.27,0.72] 0.15*** -0.08  -3.65 [0.06,0.42] 
Z 0.78 -0.29 -0.65 [0.38,1.63] 0.9 -0.31 -0.3 [0.46,1.78] 
3 0.26* -0.15 -2.3  [0.08,0.82] 0.97 -0.61 -0.04  [0.29,3.29] 
4 0.65 -0.19 -1.44 [0.37,1.17] 0.68 -0.51 -0.52 [0.16,2.92] 
5 0.81 -0.26 -0.65 [0.43,1.52] 1.46 -0.58 0.95 [0.67,3.18] 
6 Dany ete =1009 2,51 [1-2575.98] 2.49 -1.46 1.56 [0.79,7.83] 
7 2.04** 0.49 2.97 [1.27,3.26] 1.91 -0.67 1.85 [0.96,3.80] 
8 3.2074 * = 0.87 743 [1.88,5.43] 3.42*** -1.08 3.9 [1.84,6.34] 
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3 year 6 year 
OR SE z Cl OR SE z Cl 

9 3.32*** -08 5 [2.07,5.31] 1.72 -1.12 0.84 [0.49,6.13] 
10 0.85 -0.22 -0.62 [0.51,1.43] 0.94 -0.22 -0.26 [0.59,1.50] 
a 0.14*** -0.08 -3.33 [0.04,0.45] 0.45* -0.17. -2.11  ~=[0.22,0.95] 
12 0.28 -0.2 -1.78 [0.07,1.14] 0.24 -0.25  -1.39  [0.03,1.78] 
Transfer 

1 0.18*** -0.09 -3.35 [0.07,0.49] 0.25** -0.12 -2.95  [0.10,0.63] 
2 0.75 -0.39 -0.55 [0.27,2.08] 0.56 -0.26 -1.23 [0.22,1.41] 
3 0.17 -0.17 -1.78 [0.02,1.20] 0.38 -0.39 -0.93  [0.05,2.87] 
4 0.19* -0.14 -2.29 [0.05,0.79] 0.92 -0.68 -0.12 [0.21,3.96] 
5 0.28 -0.2. -1.79 [0.07,1.13] 1.09 -0.53 0.18 [0.42,2.82] 
6 1 (.) [1.00,1.00] 1 (.) [1.00,1.00] 
7 0.77 -0.36 -0.56 [0.31,1.93] 1.75 -0.7 1.4 [0.80,3.81] 
8 0.26 -0.26 -1.33 [0.04,1.90] 1.79 -0.71 1.46 [0.82,3.91] 
g) 0.83 -0.43 -0.35 [0.30,2.31] 1.28 -0.97 0.33 [0.29,5.68] 
10 0.19* -0.13 -2.34 [0.05,0.76] 0.71 -0.21 -1.17 [0.40,1.26] 
11 0.19* -0.14 -2.32 [0.05,0.77] 0.44 -0.19 -1.89  [0.19,1.03] 
12 1 (.) [1.00,1.00] 1 (.) [1.00,1.00] 
Transfer 

Prepared 
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3 year 6 year 
OR SE z Cl OR SE z Cl 

1 0.57 -0.17. -1.94 [0.32,1.01] 0.20** -0.1 -3.11 [0.07,0.55] 
2 0.5 -0.3 -1.16 [0.16,1.61] 0.68 -0.3 -0.88  [0.29,1.60] 
3 0.15 -0.15 -1.88 [0.02,1.09] 1 (.) [1.00,1.00] 
4 0.26* -0.15 -2.30 [0.08,0.82] 0.43 -0.44 -0.82 [0.06,3.21] 
5 0.38 -0.22 -1.66 [0.12,1.20] 1.65 -0.69 1.18 [0.72,3.76] 
6 0.51 -0.51 -0.67 [0.07,3.72] 2.28 -1.48 1.27 [0.64,8.13] 
7 0.84 -0.36 -0.41 [0.36,1.94] 1.98 -0.75 1.80 [0.94,4.15] 
8 236" -0.86 2.35 [1.15,4.84] 3.04** -1.05 3.22 [1.54,5.98] 
9 2.48** -0.8 2.82 [1.32,4.66] 0.62 -0.64 -0.47 [0.08,4.70] 
10 0.34*  -0.17  -2.11 [0.12,0.92] 0.95 0.25 -0.21 [0.56,1.60] 
11 1 (.) [1.00,1.00] 0.22** -0.13  -2.58 [0.07,0.69] 
a 0.24 -0.25  -1.39 [0.03,1.77] 1 (.) [1.00,1.00] 
30 Units 

1 0.26*** -0.05 -6.98 [0.18,0.38] 0.14*** -0.05 -5.63  [0.07,0.28] 
2 0.91 -0.22 -0.38 [0.58,1.45] 1.11 -0.27 0.42 [0.69,1.79] 
3 0.25*** -0.09 -3.72 [0.12,0.52] 0.72 -0.36 -0.66 [0.27,1.92] 
4 0.76 -0.14 -1.49 [0.53,1.09] 0.79 -0.40 -0.46 [0.29,2.14] 
5 1.04 -0.21 0.2 [0.71,1.54] 0.77 -0.28 -0.71 [0.38,1.57] 


Academy for College Excellence: Mixed-Methods Analysis of Long-Term Outcomes 
The RP Group | March 2018 | Page 64 


3 year 6 year 


OR SE z Cl OR SE z Cl 
6 1.84 -0.62 1.82 [0.95,3.57] 3.87** -1.97 2.66 [1.43,10.50] 
7 2.27*** -0.40 4.68 [1.61,3.21] 1.92* “0.55 2.27 [1.09,3.37] 
8 2.13** -0.49 3.28 [1.36,3.34] 2.48** -0.72 3.13 [1.40,4.37] 
2 V84**) 9-039) 2.91, [1222.77], 1.64 -0.84 0.97 [0.60,4.48] 
10 1.01 -0.17. 0.05 [0.73,1.40] 1.3 0.22 1.57 [0.94,1.80] 
11 0.12*** -0.05 -5.49 [0.06,0.26] 0.33*** -0.10 -3.77 [0.19,0.59] 
12 0.37* -0.15 -2.51 [0.17,0.80] 0.56 0.27 = =-1.19 [0.21,1.46] 


Note. *ps.05, **ps.01, ***p <.001. OR = Odds Ratio. SE = Standard Error. Z = Z-score. CI = Confidence Interval. 
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Appendix D: EDD Match Rates 


Table D1. EDD Match Rates by Year and Target Student Populations 


Years since Non-CTE Non-CTE Pre-ACE Pre-ACE ACE ACE 
participation ACE ACE nursing nursing nursing nursing 
in ACE students control students matched students students 
students peers matched 
peers 
-1 (1,228) (1,587) (369 ) (166) (307) (163) 
44.7% 57.8% 73.9% 70.3% 61.6% 69.0% 
1 (1,549) (1,859) (372) (169) (287) (172) 
56.4% 67.7% 74.5% 71.8% 57.6% 73.0% 
2 (1,739) (1,963) (373) (173) (247) (180) 
63.3% 71.5% 74.7% 73.2% 49.5% 76.4% 
3 (2,014) (2,187) (375) (180) (499) (188) 
73.3% 79.6% 75.1% 76.4% 100.0% 79.7% 
4 (1,705) (2,187) (387) (157) (337) (171) 
62.1% 66.0% 77.5% 66.6% 67.5% 72.4% 
5 (1,871) (2,013) (399) (166) (440) (184) 
68.1% 73.3% 79.9% 70.5% 88.1% 77.8% 
6 (1,890) (1,981) (413) (159) (419) (171) 
68.8% 72.1% 82.7% 67.4% 84.1% 72.6% 
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